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Congressional  Committees 

Defense  Transportation:  Air  Force’s  Airlift  Study  Met  Mandate  Requirements 

In  January  2012,  the  Department  of  Defense  (DOD)  issued  strategic  guidance  that  called  for 
recalibrating  joint  force  capabilities  and  making  selective  additional  joint  force  investments  in 
order  to  succeed  in  10  enumerated  mission  areas.1  The  guidance  also  noted  that  the  balance 
between  available  resources  and  the  nation’s  security  needs  has  never  been  more  delicate. 
DOD  initiated  a  budget-reduction  initiative  in  fiscal  year  2013  to  reduce  the  department’s  budget 
by  $486.9  billion  below  its  fiscal  year  2012  level  by  fiscal  year  2021 .  As  part  of  that  budget- 
reduction  initiative,  DOD  planned  to  retire  C-23  cargo  planes,  which  are  used  by  National  Guard 
units  both  in  their  federal  role  as  combat  units  and  in  their  state  role  as  first  responders  to 
natural  disasters.  The  decision  to  retire  these  planes  led  to  congressional  committee  interest  in 
the  adequacy  of  airlift  to  support  military  operations. 

Section  112  of  the  National  Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2012  required 
that  the  Secretary  of  the  Air  Force  conduct  a  study  to  determine  the  number  of  fixed-wing 
(plane)  and  rotary-wing  (helicopter)  aircraft  necessary  to  support  the  following  four  missions 
under  Titles  10  and  32  of  the  U.S.  Code — (1)  homeland  defense,  (2)  time-sensitive  direct 
support,  (3)  disaster  response,  and  (4)  humanitarian  assistance — at  the  following  five  levels  of 
operational  risk:  low,  medium,  moderate,  high,  and  very  high.2  Section  112  also  required  that 
the  study  be  completed  in  consultation  with  the  Secretary  of  the  Army,  the  Director  of  the 
National  Guard  Bureau,  each  supported  commander  of  a  combatant  command,  and  the 
Administrator  of  the  Federal  Emergency  Management  Agency  (FEMA).3  Additionally,  the 
Secretary  of  the  Air  Force  was  to  submit  a  report  containing  the  study  to  the  congressional 
defense  committees.  According  to  the  Air  Force,  the  Air  Force  tasked  Air  Mobility  Command  to 
perform  the  airlift  study  and  tasked  Headquarters,  Air  Force,  Analyses,  Assessments,  and 
Lessons  Learned  (A9)  with  finalizing  the  subsequent  report  based  on  the  study.  The  Air  Force 
entered  into  a  contract  with  RAND  to  conduct  the  airlift  study.4  RAND  completed  this  classified 


department  of  Defense,  Sustaining  U.S.  Global  Leadership:  Priorities  for  21st  Century  Defense  (January  2012).  The 
1 0  enumerated  mission  areas  are:  (1 )  counter  terrorism  and  irregular  warfare;  (2)  deter  and  defeat  aggression;  (3) 
project  power  despite  anti-access/area  denial  challenges;  (4)  counter  weapons  of  mass  destruction;  (5)  operate 
effectively  in  cyberspace  and  space;  (6)  maintain  a  safe,  secure,  and  effective  nuclear  deterrent;  (7)  defend  the 
homeland  and  provide  support  to  civil  authorities;  (8)  provide  a  stabilizing  presence;  (9)  conduct  stability  and 
counterinsurgency  operations;  and  (10)  conduct  humanitarian,  disaster  relief,  and  other  operations. 

2See  Pub.  L.  No.  112-81,  §  112(d)  (2011). 

3DOD’s  nine  combatant  commands  are:  (1)  U.S.  Africa  Command,  (2)  U.S.  Central  Command,  (3)  U.S.  European 
Command,  (4)  U.S.  Northern  Command,  (5)  U.S.  Pacific  Command,  (6)  U.S.  Southern  Command,  (7)  U.S.  Special 
Operations  Command,  (8)  U.S.  Strategic  Command,  and  (9)  U.S.  Transportation  Command. 

4RAND  operates  a  federally  funded  research  and  development  center  that  provides  the  Air  Force  with  studies  and 
analyses  through  a  program  called  Project  Air  Force.  Air  Mobility  Command,  through  the  Project  Air  Force  contract, 
funded  RAND  to  conduct  the  NDAA  for  Fiscal  Year  2012  Air  Force  Airlift  Study. 
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study  in  December  2012.  The  Air  Force  submitted  its  classified  executive  summary  report, 
which  included  the  RAND  study,  to  the  congressional  defense  committees  in  March  2013. 

Section  112  also  included  a  provision  for  GAO  to  conduct  a  sufficiency  review  of  the  study.5 
This  report  determines  the  extent  to  which  the  Air  Force’s  airlift  study  conformed  to  generally 
accepted  research  standards  and  addressed  the  mandate. 

To  determine  the  extent  to  which  the  Air  Force’s  airlift  study  conformed  to  generally  accepted 
research  standards,  we  identified  applicable  generally  accepted  research  standards — on 
design,  execution,  and  presentation — using  relevant  prior  GAO  reports,  and  criteria  that  define  a 
sound  and  complete  study.  Two  analysts  and  two  social  scientists:  (1)  independently  reviewed 
the  airlift  study  and  plan  to  analyze  the  information  against  the  identified  generally  accepted 
research  standards,  (2)  determined  whether  the  study  conformed  to  the  standards,  and  (3) 
reconciled  the  four  analyses  into  one  complete  analysis.  See  enclosure  I  for  an  outline  of  the 
generally  accepted  research  standards  we  used.  We  also  met  with  the  officials  who  conducted 
the  study  to  discuss  the  research  standards  they  had  used  as  well  as  the  study’s  design, 
execution,  and  presentation.  To  determine  the  extent  to  which  the  Air  Force’s  airlift  study 
addressed  the  mandate,  we  reviewed  and  analyzed  the  requirements  for  the  study  outlined  in 
Section  1 1 2  of  the  NDAA  for  Fiscal  Year  201 2  and  compared  these  requirements  with  the 
information  contained  in  the  Air  Force  study.  Specifically,  we  conducted  an  analysis  to 
determine  whether  the  Air  Force  airlift  study  fully  met,  partially  met,  or  did  not  meet  the 
mandate's  requirements.  We  also  interviewed  the  officials  who  led  the  study  from  the  Air  Force, 
including  the  Air  Mobility  Command  and  RAND  as  well  as  officials  identified  to  be  the  most 
knowledgeable  from  the  Air  Force,  Army,  National  Guard  Bureau,  combatant  commands,  and 
FEMA,  to  determine  whether  they  or  their  offices  had  been  consulted  during  the  completion  of 
the  study  and  what  other  observations,  if  any,  they  had  about  the  study  or  conduct.  See 
enclosure  II  for  additional  information  about  our  scope  and  methodology. 

We  conducted  this  performance  audit  from  June  2014  to  May  2015  in  accordance  with  generally 
accepted  government  auditing  standards.  Those  standards  require  that  we  plan  and  perform  the 
audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objective.  We  believe  that  the  evidence  obtained  provides  a 
reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 

Results  in  Brief 

We  found  that  the  Air  Force  airlift  study  conformed  to  generally  accepted  research  standards 
and  generally  met  the  requirements  of  the  mandate  related  to  the  study  by  fully  addressing  the 
three  elements  required.  For  instance,  we  found  that  the  design  standard  was  fulfilled  because 
the  airlift  study  had  a  plan  that  was  methodical,  and  that  presented  the  study’s  tasks,  timelines, 
and  deliverables.  We  also  found  that  the  assumptions  and  constraints  of  the  plan  were  identified 
and  deviations  from  the  plan  were  mentioned,  such  as  the  rationale  of  selecting  one  domestic 
scenario  to  discuss  two  missions.  Additionally,  we  found  that  the  study  fully  addressed  the 
requirements  of  the  mandate  in  that  it 

•  analyzed  the  four  missions  identified  using  four  approved  scenarios:  two  overseas 
scenarios  for  the  time-sensitive,  direct  support  mission;  one  domestic  scenario  for  the 


5See  §  112(e). 
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homeland  defense  mission;  and  one  additional  domestic  scenario  for  both  the  disaster 
response  and  humanitarian  assistance  missions; 

•  generated  the  requirements  at  five  risk  levels — very  high,  high,  moderate,  medium,  and 
low — with  each  risk  level  identifying  the  likelihood  of  meeting  airlift  demand  with  a  given 
deployed  fleet  size;  and 

•  was  conducted  in  consultation  with  representatives  from  the  Army,  the  National  Guard 
Bureau,  the  combatant  commands,  and  FEMA,  which  we  determined  through  reviewing 
the  stakeholders’  written  comments  on  the  report  and  interviewing  officials. 

While  most  of  the  stakeholders  agreed  with  the  study’s  conclusions,  many  of  them  did  so  with 
caveats.  For  instance,  officials  from  the  Army,  U.S.  European  Command,  and  U.S. 
Transportation  Command  noted  that  although  their  respective  organizations  concurred  with  the 
airlift  study  overall,  they  questioned  the  wider  utility  of  the  study  because  it  did  not  consider 
planning  limitations  such  as  multiple  simultaneous  contingencies,  or  take  into  consideration 
unavailable  aircraft. 

We  are  not  making  any  recommendations  in  this  report. 

Airlift  Study  Conformed  to  Generally  Accepted  Research  Standards  and  Met  the 
Requirements  of  the  Mandate 

The  Airlift  Study  Conformed  to  Generally  Accepted  Research  Standards 

The  airlift  study  generally  followed  the  research  standards  we  derived  from  previous  GAO  work, 
other  research  literature,  and  DOD  guidance.6  We  evaluated  the  airlift  study  against  a  checklist 
of  these  standards  that  measures  the  design,  execution,  and  presentation  of  research  studies.7 
We  also  conducted  interviews  with  those  officials  from  RAND,  Air  Mobility  Command,  and  Air 
Force  A9  who  worked  on  the  study  to  discuss  the  standards  utilized  in  the  study.  Four 
independent  analyses  were  completed  and  then  reconciled  to  determine  the  extent  to  which  the 
airlift  study  conformed  to  three  overarching  standards.  Each  of  the  three  standards  included 
components  that  determined  whether  the  standard  was  met.  In  instances  where  evidence 
determined  that  components  of  a  standard  were  not  applicable,  the  standard  was  still 
determined  to  be  met.  For  instance,  while  an  official  explained  that  the  study  did  not  include 
baseline  data,  a  component  of  the  execution  standard,  the  data  utilized  for  the  analyses  were 
approved  and  vetted.  Table  1  provides  a  summary  of  the  assessment  of  the  airlift  study  against 
the  generally  accepted  research  standards. 


6As  is  the  case  for  most  studies,  there  were  some  limitations,  which  we  describe  in  the  last  section  of  this  report.  We 
considered  these  limitations  in  our  assessment  but,  based  on  the  preponderance  of  the  evidence,  determined  that  the 
applicable  standards  were  generally  met. 

7We  assessed  those  generally  accepted  standards  that  were  relevant  for  this  type  of  study. 
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Table  1 :  GAO’s  Assessment  of  the  Extent  to  Which  the  Air  Force  Airlift  Study  Conformed 
with  Generally  Accepted  Research  Standards 


Generally  accepted 
research  standards 

Comments 

Design 

The  design  plan  was  coordinated  with  the  project’s  sponsor  to 
ensure  that  the  research  was  correctly  focused  and  planned. 
Assumptions  and  constraints  of  the  plan  were  identified,  and 
deviations  from  the  plan  were  mentioned,  such  as  the  rationale 
of  selecting  one  domestic  scenario  to  discuss  two  missions. 

Execution 

The  execution  of  the  airlift  study  was  consistent  with  the  study 
plan  and  schedule.  The  study  utilized  sufficiently  reliable  data — 
specifically,  flight  data  approved  by  the  study’s  stakeholders  for 
use  in  estimating  ranges  of  mission  success  when  utilizing 
certain  amounts  of  fixed-  and  rotary-wing  aircraft.  RAND  used 
these  data  to  conduct  Monte  Carlo  analyses,  which  are  an 
appropriate  analytical  tool  for  modeling  risk  levels  for  mission 
accomplishment  given  various  amounts  and  types  of  airlift 
support. 

Presentation 

The  study’s  objectives,  assessment,  and  conclusions  were  well 
documented  and  supported  by  the  analysis  contained  in  the 
study,  and  the  stakeholders  were  informed  of  the  study’s  results. 

Source:  GAO  analysis  of  the  Air  Force  data.  |  GAO-1 5-457R 


The  generally  accepted  research  standards  we  used  to  evaluate  the  airlift  study  are  included  in 
enclosure  I. 

The  Airlift  Study  Met  the  Requirements  of  the  Mandate 

We  found  that  the  airlift  study  met  the  requirements  of  Section  1 12(d)  of  the  NDAA  for  Fiscal 
Year  201 2.  The  mandate  required  the  Secretary  of  the  Air  Force,  in  consultation  with  the 
Secretary  of  the  Army,  the  Director  of  the  National  Guard  Bureau,  each  supported  combatant 
commander,  and  the  Administrator  of  FEMA,  to  conduct  a  study  to  determine  the  number  of 
fixed-wing  and  rotary-wing  aircraft  required  to  support  four  specified  missions  under  Titles  10 
and  32,  U.S.  Code — homeland  defense,  time-sensitive  direct  support,  disaster  response,  and 
humanitarian  assistance — at  five  levels  of  operational  risk.8  Our  analysis  of  the  extent  to  which 
the  RAND  study  met  the  requirements  of  the  mandate  is  summarized  in  table  2  below. 


8See  Pub.  L.  No.  112-81,  §  112(d). 
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Table  2:  Summary  of  Our  Assessment  of  the  Air  Force  Airlift  Study  in  Response  to 
Section  112  of  the  National  Defense  Authorization  Act  for  Fiscal  Year  2012 


Study  requirement 

Our  assessment 
of  the  Air  Force 
study 

Comments 

(1)  Determine  the  number  of 
fixed-wing  and  rotary-wing 
aircraft  required  to  support  the 
following  missions  (at  the 
various  risk  levels  mentioned 
below): 

•  Homeland  defense 

•  Time-sensitive,  direct 
support  to  forces  consisting 
of  the  regular  component  of 
the  Army  and  the  National 
Guard 

•  Disaster  response 

•  Humanitarian  assistance 

Addressed 

The  airlift  study  analyzed  the  four 
missions  using  four  approved  scenarios: 
two  overseas  scenarios  for  the  time- 
sensitive,  direct  support  mission;  one 
domestic  scenario  for  the  homeland 
defense  mission;  and  one  additional 
domestic  scenario  for  both  the  disaster 
response  and  humanitarian  assistance 
missions. 

(2)  Determine  the  number  of 
fixed-and  rotary-wing  aircraft 
required  to  support  the 
missions  at  the  following 
operational  risk  levels:  Low, 
Medium,  Moderate,  High,  Very 
High 

Addressed 

Although  DOD  generally  recognizes  four 
levels  of  risk,  to  comply  with  the  mandate 
requirement  the  risks  were  modified  into 
five  risk  levels — very  high,  high,  moderate, 
medium,  and  low — in  the  Air  Force  study. 
Each  risk  level  identifies  the  likelihood  of 
meeting  airlift  demand  with  a  given 
deployed  fleet  size. 

(3)  Perform  the  airlift  study  in 

consultation  with 

•  Secretary  of  the  Army 

•  Director  of  National  Guard 
Bureau 

•  Each  supported 
commander  of  a  combatant 
command 

•  Administrator  of  the  Federal 
Emergency  Management 
Agency  (FEMA) 

Addressed 

The  Air  Force  consulted  with  officials  from 
the  Army,  the  National  Guard  Bureau,  the 
combatant  commands,  and  FEMA.  We 
verified  participation  through  written 
documentation  provided  from  the  Air 

Force  and  RAND  and  interviews  and 
correspondence  with  the  stakeholders. 

Source:  GAO  analysis  of  Air  Force  data. 

|  GAO-1 5-457R 

Missions  and  Scenarios 


To  analyze  the  four  missions  specified  in  Section  112,  officials  selected  a  representative 
scenario  of  each  of  the  mission  areas  that  portrayed  either  (1)  time-sensitive  direct  support  to 
combat  forces  on  the  ground  outside  of  the  continental  United  States  or  (2)  civil  support 
missions  within  the  continental  United  States.  To  analyze  the  fixed-  and  rotary-wing  airlift 
requirements  for  the  time-sensitive,  direct  support  mission,  officials  used  two  warfighting 
scenarios  that  placed  various  demands  on  airlift.  For  the  other  three  missions  specified  in  the 
provision,  officials  chose  domestic  scenarios  from  a  set  of  26  approved  planning  scenarios  and 
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variants  that  involved  a  combination  of  humanitarian  assistance,  disaster  response,  and 
homeland  defense.9  Officials  then  further  organized  these  domestic  scenarios  into  seven 
groups — for  example,  one  group  considered  nuclear  attacks,  and  another  considered  wide-area 
casualties  and  infrastructure  damage,  including  an  earthquake  and  hurricane — and  determined 
the  various  types  of  support  that  would  be  required  for  each  group.  Based  on  this  assessment, 
officials  concluded  that  wide-area  catastrophes,  such  as  a  natural  disaster  or  nuclear  attack, 
would  be  the  most  demanding  events  for  airlift.  Consequently,  officials  elected  two  scenarios — a 
10-kiloton  nuclear  terrorist  attack  in  Washington,  D.C.,  and  a  large  (7.7  magnitude)  earthquake 
in  the  central  United  States — to  analyze  the  homeland  defense,  humanitarian  assistance,  and 
disaster  response  missions. 

While  the  same  domestic  scenario  was  used  for  the  analysis  of  both  the  disaster  response  and 
humanitarian  assistance  missions,  according  to  Army  and  Air  Force  officials,  as  well  as  officials 
from  FEMA  and  the  National  Guard  Bureau,  humanitarian  assistance  is  generally  recognized  as 
assistance  provided  to  other  nations.  Because  of  this,  some  stakeholders  expressed  a 
preference  for  using  a  foreign  scenario  to  conduct  the  analysis  of  airlift  requirements  associated 
with  a  humanitarian  assistance  mission.10  When  asked  why  the  same  domestic  scenario  was 
used  to  analyze  both  the  disaster  response  and  humanitarian  assistance  missions,  RAND 
officials  stated  that  doing  so  would  enable  them  to  analyze  airlift  requirements  in  both  Title  10 
and  Title  32  statuses.  Moreover,  a  RAND  official  stated  that  tasks  generally  associated  with 
humanitarian  assistance  missions  would  remain  the  same  whether  the  scenario  used  to  analyze 
the  mission  was  domestic  or  foreign.  The  study’s  stakeholders  agreed  that  using  a  domestic 
scenario  to  analyze  the  humanitarian  assistance  missions  was  adequate  for  the  purposes  of  the 
study. 


Requirements  per  Risk  Level 

To  determine  the  requirements  of  fixed-  and  rotary-wing  aircraft  for  each  mission  at  each  of  the 
five  risk  levels,  officials  categorized  each  level  by  the  likelihood  of  meeting  airlift  demand  with  a 
given  deployed  fleet  size.  For  the  airlift  study,  RAND  interpreted  DOD’s  risk  assessment 
definition  of  probability  to  be  four  levels  of  risk:  (1)  high — unlikely  (0-20  percent)  to  meet  airlift 
demand  at  a  given  deployed  fleet  size,  (2)  significant — questionable  (20-50  percent),  (3) 
moderate — likely  (50-80  percent),  and  (4)  low — very  likely  (80-100  percent).  However,  to 
comply  with  the  mandate  requirement,  the  risk  scale  in  the  study  was  expanded  to  include  a  fifth 
level  between  the  levels  of  significant  and  moderate,  and  the  thresholds  were  adjusted  to  (1) 
high — unlikely  (0-20  percent),  (2)  significant — questionable  (20-40  percent),  (3)  moderate- 
significant — likely  to  questionable  (40-60  percent),  (4)  moderate — likely  (60-80  percent),  and 


9Doctrinally,  response  to  disasters  in  the  United  States  homeland  may  be  classified  as  “defense  support  of  civil 
authorities.”  See  generally  Joint  Chiefs  of  Staff,  Joint  Pub.  3-28,  Defense  Support  of  Civil  Authorities  (July  31 , 2013). 
The  26  approved  planning  scenarios  constitute  a  list  developed  from  the  Federal  Interagency  Community,  including 
DOD  and  FEMA,  for  use  in  national,  federal,  state,  and  local  homeland-preparedness  activities. 

1 0 

Various  statutes  and  DOD  guidance  similarly  discuss  humanitarian  assistance  provided  outside  the  United  States. 
See,  e.g.,  10  U.S.C.  §  401  (providing  for  humanitarian  and  civic  assistance  activities  in  conjunction  with  military 
operations  in  a  foreign  country);  10  U.S.C.  §  402  (providing  for  transportation  of  relief  supplies  for  humanitarian 
assistance);  10  U.S.C.  §  2561  (providing  for  transportation  of  humanitarian  relief  and  other  humanitarian  assistance 
worldwide);  Department  of  Defense  Instruction  2205.02,  Humanitarian  and  Civic  Assistance  (HCA)  Activities  (June 
23,  2014)  (implementing  10  U.S.C.  §  401);  Department  of  Defense  Instruction  3000.05,  Stability  Operations  (Sept. 
16,  2009)  (discussing  humanitarian  assistance  and  other  missions);  Joint  Chiefs  of  Staff,  Joint  Pub.  3-29,  Foreign 
Humanitarian  Assistance  (Jan.  3,  2014). 
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(5)  low — very  likely  (80-100  percent)  to  meet  airlift  demand  at  a  given  deployed  fleet  size.11  The 
study  determined  the  number  of  generic  fixed-  and  rotary-wing  aircraft  required  to  meet  the 
demands  of  the  scenarios  listed  in  the  study. 

Consultation  with  Stakeholders 

The  Air  Force  consulted  with  all  of  the  stakeholders  listed  in  the  mandate,  and  most 
stakeholders  concurred  with  the  airlift  study.  However,  some  of  the  stakeholders  identified 
caveats  and  expressed  differing  views  of  the  study’s  analysis  and  utility.  For  example, 
stakeholders  noted  the  study’s  limited  scope  and  therefore  its  usefulness  for  prescribing  total 
airlift  requirements;  the  fact  that  the  study  did  not  address  airlift  requirements  associated  with 
emerging  missions;  and  the  specific  domestic  scenarios  selected  for  analysis.  Moreover, 
although  the  Air  Force  supported  RAND’s  analysis,  it  had  a  different  opinion  regarding  the 
requirement  for  domestic  disasters,  and  adjusted  the  requirement  for  fixed-wing  assets  in 
domestic  disasters  in  its  report  accompanying  the  RAND  study.  Specifically,  RAND  had 
concluded  that  fixed-wing  assets  would  be  difficult  to  land  in  unstable  areas  in  disaster  zones, 
and  therefore  it  did  not  include  a  requirement  for  fixed-wing  assets  in  its  earthquake  scenario. 

Air  Force  officials,  however,  asserted  that  fixed-wing  aircraft  flying  into  landing  areas  outside  the 
disaster  area  could  be  used  to  bring  in  equipment  that  would  then  be  transported  by  other 
means  to  the  disaster  area.  Because  of  this,  the  airlift  report  amended  the  airlift  study’s 
conclusions  about  the  airlift  requirements  for  domestic  disaster  scenarios  by  adding  a  fixed-wing 
requirement. 

The  Army,  FEMA,  and  eight  of  the  nine  combatant  commands  concurred  with  the  airlift  study, 
and  a  few  noted  caveats  in  their  written  comments  to  the  airlift  study.  U.S.  Southern  Command 
and  the  National  Guard  Bureau  did  not  concur  with  the  study.  Specifically,  officials  from  the 
Army,  U.S.  European  Command,  and  U.S.  Transportation  Command  stated  that  although  their 
respective  organizations  concurred  with  the  airlift  study  overall,  they  questioned  the  wider  utility 
of  the  study  because  it  did  not  consider  planning  limitations  such  as  multiple  simultaneous 
contingencies,  or  take  into  consideration  unavailable  aircraft  (for  example,  aircraft  that  are  in 
depots,  are  used  for  training,  or  are  used  for  backup  support).  Further,  officials  from  U.S. 
Southern  Command  did  not  concur  with  the  study  because  the  study  did  not  address  the 
emerging  requirements  associated  with  DOD  providing  increased  support  to  U.S.  diplomatic 
facilities  and  personnel  overseas.  In  its  written  comments,  a  command  official  noted  that  a 
scenario  addressing  these  emerging  airlift  requirements  should  have  been  considered  to 
address  the  time-sensitive  direct  support  and  disaster  relief  missions,  because  it  would  likely 
generate  a  requirement  for  airlift  both  into  and  out  of  the  affected  area.  Both  RAND  and  the  Air 
Force  acknowledged  that  analysis  of  such  a  scenario  could  be  important,  but  noted  that  DOD 
does  not  currently  have  a  representative  scenario  for  this  type  of  event.  Further,  Air  Force 
officials  stated  that  when  planning  for  airlift  requirements,  to  maintain  transparency  and  equality, 
representative  scenarios  are  selected  from  a  preapproved  list  to  which  all  services  have  access, 
and  it  is  not  generally  acceptable  to  use  scenarios  outside  of  this  list  for  planning. 

In  addition  to  U.S.  Southern  Command,  officials  from  FEMA  and  the  National  Guard  Bureau 
raised  concerns  with  the  scenarios  selected.  Specifically,  FEMA  officials  noted  that  the 
information  RAND  used  while  analyzing  the  domestic  scenarios  it  selected  was  not  up  to  date. 
For  instance,  FEMA  stated  that  the  earthquake  scenario  did  not  incorporate  lessons  learned 
that  FEMA  had  previously  identified.  According  to  a  FEMA  official,  this  information  was  offered 


^According  to  the  airlift  study,  risk  level  probability  ranges  are  rounded.  Each  range  is  inclusive  of  the  lower  bound 
shown  but  not  inclusive  of  the  upper  bound. 
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to  RAND.  However,  a  RAND  official  stated  that  the  data  used  were  the  most  current  information 
made  available  at  the  time  of  the  analysis.  Similarly,  officials  from  the  National  Guard  Bureau 
questioned  RAND’s  decision  not  to  use  a  hurricane  scenario  to  analyze  airlift  requirements 
associated  with  a  disaster  response  mission.  According  to  these  officials,  a  hurricane  places 
stressors  on  airlift  that  would  not  be  captured  by  analyzing  earthquakes.  As  a  result,  the 
National  Guard  Bureau  did  not  concur  with  RAND’s  determination  that  there  are  sufficient  airlift 
resources  to  meet  requirements.12  According  to  RAND  officials,  they  did  not  choose  a  large 
hurricane  scenario  like  Hurricane  Katrina  primarily  because  the  timing  and  location  of  hurricanes 
can  be  anticipated,  thus  allowing  for  pre-event  preparations  that  would  greatly  reduce  airlift 
requirements.  For  example,  evacuations  in  areas  in  or  near  a  hurricane’s  anticipated  path  prior 
to  a  hurricane  making  landfall  would  reduce  the  requirement  for  search  and  rescue  and  for 
medical  evacuation  airlift.  While  the  Air  Force  supported  RAND’s  observation  that  airlift 
requirements  for  a  hurricane  scenario  can  be  predetermined  and  therefore  mitigated,  in  the 
airlift  report  accompanying  the  study  the  Air  Force  still  included  an  assessment  of  a  hurricane 
scenario  because  of  its  potential  stressors  on  airlift. 

Agency  Comments 

We  provided  a  draft  of  this  report  to  DOD  for  review  and  comment.  In  written  comments,  which 
are  reprinted  in  enclosure  III,  DOD  concurred  with  our  findings. 

Additionally,  we  provided  a  draft  of  this  report  to  DHS  for  review  and  comment.  DHS  did  not 
have  any  comments. 


We  are  sending  copies  of  this  report  to  the  appropriate  congressional  committees,  the  Secretary 
of  Defense,  the  Secretaries  of  the  Air  Force  and  Army,  the  Chief  of  the  National  Guard  Bureau, 
and  the  Administrator  of  FEMA.  In  addition,  the  report  is  available  at  no  charge  on  the  GAO 
website  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me  at  (202)  512-5431  or 
russellc@gao.gov.  Contact  points  for  our  Offices  of  Congressional  Relations  and  Public  Affairs 
may  be  found  on  the  last  page  of  this  report.  GAO  staff  who  made  major  contributions  to  this 
report  include  Guy  LoFaro  (Assistant  Director),  Kim  Seay  (Assistant  Director),  Martin  De  Alteriis, 
Randy  DeLeon,  Mike  Shaughnessy,  Anne  Stevens,  Cheryl  Weissman,  and  Natasha  Wilder. 


Cary  Russell 

Director,  Defense  Capabilities  and  Management 
Enclosures  -  3 


12During  the  same  period  the  RAND  study  was  being  conducted,  the  National  Guard  Bureau  conducted  an 
unclassified  study — which  included  a  hurricane  scenario  analysis — that ,  by  contrast,  found  that  the  Air  National 
Guard  would  not  be  able  to  meet  the  anticipated  2017  force  structure  requirements.  National  Guard  Bureau, 
Functional  Solutions  Analysis  for  Fixed/Rotary  Wing  Support  to  Homeland  Defense/Defense  Support  to  Civil 
Authorities  (HD/DSCA)  and  State-Level  Missions  (Nov.  2,  2012). 
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List  of  Committees 


The  Honorable  John  McCain 
Chairman 

The  Honorable  Jack  Reed 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Thad  Cochran 
Chairman 

The  Honorable  Richard  J.  Durbin 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Rodney  Frelinghuysen 
Chairman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Enclosure  I:  Generally  Accepted  Research  Standards  Checklist 

We  applied  the  applicable  standards  derived  from  relevant  GAO  work,  literature,  and  guidance 
that  we  identified  as  relevant  to  the  Air  Force  airlift  study,  as  shown  in  table  2. 

Table  3:  GAO  Generally  Accepted  Research  Standards  Checklist  for  the  Air  Force  Airlift 

Study _ 

Design:  The  study  is  well  designed _ 

I.  Study,  plan  scope,  and  objectives _ 

_ l.a  Was  the  study  plan  followed? _ 

_ l.b  Are  the  objectives  clearly  stated? _ 

_ l.c  Were  the  deviations  from  the  plan  explained  and  documented? _ 

_ l.d  Is  the  study’s  scope  clearly  defined? _ 

II.  Assumptions  and  constraints  are  reasonable  and  consistent _ 

_ ll.a  Are  the  assumptions  explicitly  identified? _ 

_ ll.b  Are  the  study  assumptions  necessary  and  reasonable? _ 

_ ll.c  Are  the  major  constraints  identified  and  discussed? _ 

_ I  l.d  Do  the  study  assumptions  support  a  sound  analysis? _ 

_ ll.e  Are  the  assumptions  used  in  analyses  common  throughout  the  study  and  models? 

_ ll.f  Are  the  assumptions  varied  to  allow  for  sensitivity  analyses? _ 

_ ll.g  Do  the  assumptions  contribute  to  an  objective  and  balanced  research  effort? _ 

III.  Scenarios  and  threats  are  reasonable _ 

_ 1 1 1  .a  Are  scenarios  traceable  back  to  formal  guidance? _ 

_ lll.b  Do  scenarios  represent  a  reasonably  complete  range  of  conditions? _ 

_ lll.c  Were  the  threats  varied  to  allow  for  the  conduct  of  sensitivity  analysis? _ 

Execution:  The  study  is  well  executed _ 

IV.  Methodology  is  successfully  executed _ 

IV. a  Was  the  study  methodology  executed  consistently  with  the  (airlift  requirements) 

_ study  plan  and  schedule? _ 

IV. b  Does  the  methodology  support  accomplishing  the  objectives  presented  in  the  study 
_ plan? _ 

IV.  c  Were  the  models  used  to  support  the  analyses  adequate  for  their  intended 

_ purpose? _ 

_ IV.d  Were  the  model  input  data  properly  generated  to  support  the  methodology? _ 

V.  (Analytical)  Baseline  data  and  other  data  used  to  support  study  and  analyses  validated, 

verified,  and  approved3 _ 

V. a  Is  the  (analytical)  baseline  fully  and  completely  identified  and  used  consistently 

_ throughout  the  study  for  the  various  analyses?3 _ 

_ V.b  Were  data  limitations  identified  and  was  the  effect  of  the  limitations  fully  explained? 

_ V.c  Were  the  baseline  data  verified  and  validated?3 _ 

_ V.d  Was  the  data  verification  and  validation  process  documented? _ 

VI.  Models,  simulations,  and  verification,  validation,  and  accreditation  (VV&A)  are 

reasonable3 _ 

VI.  a  Was  a  VV&A  report  that  addresses  the  models  and  data  certification  included  in 

_ the  report?3 _ 

_ VI. b  Were  modeling  and  simulation  limitations  identified  and  explained? _ 

_ VI  .c  Has  each  model  in  the  study  been  described? _ 

_ VI .d  Are  the  model  processes  clearly  explained,  documented,  and  understood? _ 

VII.  Measures  of  effectiveness  (MOE)  and  essential  elements  of  analysis  (EEA)  are _ 
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addressed?3 _ 

_ VI I  .a  Do  MOEs  adhere  to  the  guidance  in  the  study  terms  of  reference?3 _ 

_ VI I .b  Are  the  MOEs  fully  addressed  in  the  study?3 _ 

_ VI I  .c  Are  the  EEAs  addressed  in  the  study?3 _ 

Presentation  of  results:  Timely,  complete,  accurate,  concise,  and  relevant  to  the  client 

and  stakeholders _ 

VIII.  Presentation  of  results  supports  findings _ 

_ Vlll.a  Does  the  report  address  the  objectives? _ 

Vlll.b  Does  the  report  present  an  assessment  that  is  well  documented  and  conclusions 

_ that  are  supported  by  the  analyses? _ 

_ Vlll.c  Are  the  conclusions  sound  and  complete? _ 

_ Vlll.d  Are  recommendations  supported  by  analyses?3 _ 

_ Vlll.e  Is  a  realistic  range  of  options  provided? _ 

_ Vlll.f  Are  the  study  results  presented  in  the  report  in  a  clear  manner? _ 

_ Vlll.g  Are  study  participants  informed  of  the  study  results? _ 

Source:  GAO.  |  GAO-1 5-457R 

aDenotes  components  deemed  not  applicable. 


Page  1 1 


GAO-1 5-457R  Defense  Transportation 


Enclosure  II:  Scope  and  Methodology 

To  determine  the  extent  to  which  the  Air  Force  airlift  study  conformed  to  generally  accepted 
research  standards,  we  identified  generally  accepted  research  standards  and  compared  the 
study  to  those  standards.  To  identify  these  standards  we  reviewed  checklists  of  generally 
accepted  research  standards  from  prior  GAO  work  that  also  reviewed  the  Department  of 
Defense  (DOD)  mobility  requirements  studies.13  In  addition  to  reviewing  these  checklists  and 
other  research  literature  and  DOD  guidance,  we  identified  frequently  occurring,  generally 
accepted  research  standards  that  are  relevant  for  defense  studies  and  that  define  a  well- 
documented  and  clearly  presented  study.  The  checklist  we  developed  categorized  the 
standards  into  three  areas — design,  execution,  and  presentation  (see  end.  I  for  a  list  of  the 
specific  standards).  Two  analysts  and  two  social  scientists  reviewed  the  Air  Force’s  classified 
airlift  study  that  was  included  in  the  Air  Force’s  classified  airlift  executive  summary  report,  the 
airlift  report,  and  the  plan  of  the  study  and  analyzed  the  information  presented  against  the 
identified  generally  accepted  research  standards  to  determine  whether  the  study  met  the 
standard.14  Then  the  four  separate  analyses  were  reconciled  to  create  one  complete  analysis  of 
the  extent  to  which  the  study  conformed  to  the  identified  standards.  We  clarified  any 
disagreements  within  the  analyses  and,  in  some  instances,  we  determined  that  some  of  the 
standards  we  identified  in  our  checklist  were  not  applicable  to  the  airlift  study.15  Therefore,  we 
did  not  measure  the  study  against  standards  that  were  not  applicable.  We  also  interviewed 
officials  from  Air  Force  Air  Mobility  Command  and  RAND  to  discuss  their  participation  in  the 
study  and  from  Headquarters  Air  Force  to  discuss  their  knowledge  of  the  subject  matter. 

To  determine  the  extent  to  which  the  Air  Force’s  study  addressed  the  requirements  of  the 
mandate,  two  analysts  conducted  a  content  analysis  to  determine  the  extent  to  which  the  airlift 
study  met  the  requirements  in  Section  1 12  of  the  National  Defense  Authorization  Act  for  Fiscal 
Year  201 2.  We  created  a  checklist  of  the  mandate  requirements  and  compared  them  with  the 
information  contained  in  the  classified  Air  Force  airlift  study  accompanying  the  Air  Force  airlift 
report.  We  determined  whether  the  study  fully  met,  partially  met,  or  did  not  meet  the  mandate’s 
requirements.  We  also  interviewed  pertinent  officials  within  the  Air  Force,  RAND,  the  Army,  the 
National  Guard  Bureau,  the  combatant  commands,  and  the  Federal  Emergency  Management 
Agency  to  determine  their  respective  roles  and  responsibilities  in  completing  the  study. 


Specifically,  during  the  course  of  our  engagement,  we  visited  or  contacted  the  following 
organizations: 

Department  of  Defense 

•  Headquarters  Air  Force 

•  Air  Mobility  Command 


13GAO,  Defense  Transportation:  Study  Limitations  Raise  Questions  about  the  Adequacy  and  Completeness  of  the 
Mobility  Capabilities  Study  and  Report,  GAO-06-  938  (Washington,  D.C.:  Sept.  20,  2006);  Defense  Transportation: 
Additional  Information  Is  Needed  for  DOD’s  Mobility  Capabilities  and  Requirements  Study  2016  to  Fully  Address  All 
of  Its  Study  Objectives,  GAO-1 1-82R  (Washington,  D.C.:  Dec.  8,  2010). 

14 

RAND  uses  the  term  “project  description”  instead  of  the  term  “study  plan.”  Both  terms  can  be  used  to  describe  a 
document  showing  client-approved  project  or  study  timelines  and  deliverables. 

15As  is  the  case  for  most  studies,  there  were  some  limitations,  which  we  describe  in  the  last  section  of  this  report. 

We  considered  these  limitations  in  our  assessment,  but  made  our  determinations  based  on  the  preponderance  of  the 
evidence  for  each  standard. 
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•  Headquarters,  Department  of  the  Army 

•  National  Guard  Bureau 

•  U.S.  Northern  Command 

•  U.S.  European  Command 

•  U.S.  Southern  Command 

•  U.S.  Africa  Command 

•  U.S.  Central  Command 

•  U.S.  Pacific  Command 

•  U.S.  Transportation  Command 

•  U.S.  Special  Operations  Command 

•  U.S.  Strategic  Command 

Department  of  Homeland  Security 

•  Federal  Emergency  Management  Agency 

RAND  Corporation 

We  conducted  this  performance  audit  from  June  2014  to  May  2015  in  accordance  with  generally 
accepted  government  auditing  standards.  Those  standards  require  that  we  plan  and  perform  the 
audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained  provides 
a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 


Page  13 


GAO-1 5-457R  Defense  Transportation 


Enclosure  III:  Comments  from  the  Department  of  Defense 


HEADQUARTERS  UNITED  STATES  AIR  FORCE 
WASHINGTON,  DC 


DEPARTMENT  OF  THE  AIR  FORCE 


May  11,2015 


Mr.  Cary  Russell 

Director,  Defense  Capabilities  and  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington  DC  20548 

Dear  Mr.  Russell: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report 
GAO-1 5-457R,  “DEFENSE  TRANSPORTATION:  Air  Force’s  Airlift  Study  Met  Mandate 
Requirements,”  dated  April  17,  2015  (GAO  Code  351938). 

The  Department  concurs  as  written  and  appreciates  the  opportunity  to  comment  on  the 
draft  report. 


Sincerely, 


WILLIAM  L.  TROY  III,  GS-15,  DAFC 
Technical  Director,  Force  Structure  Analyses 
Headquarters  Air  Force 


(351938) 
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This  is  a  work  of  the  U.S.  government  and  is  not  subject  to  copyright  protection  in 
the  United  States.  The  published  product  may  be  reproduced  and  distributed  in 
its  entirety  without  further  permission  from  GAO.  However,  because  this  work 
may  contain  copyrighted  images  or  other  material,  permission  from  the  copyright 
holder  may  be  necessary  if  you  wish  to  reproduce  this  material  separately. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in 
meeting  its  constitutional  responsibilities  and  to  help  improve  the 
performance  and  accountability  of  the  federal  government  for  the 

American  people.  GAO  examines  the  use  of  public  funds; 
evaluates  federal  programs  and  policies;  and  provides  analyses, 
recommendations,  and  other  assistance  to  help  Congress  make 
informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at 
no  cost  is  through  GAO’s  website  (http://www.gao.gov).  Each 
weekday  afternoon,  GAO  posts  on  its  website  newly  released 
reports,  testimony,  and  correspondence.  To  have  GAO  e-mail  you 
a  list  of  newly  posted  products,  go  to  http://www.gao.gov  and 
select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages 
in  the  publication  and  whether  the  publication  is  printed  in  color  or 
black  and  white.  Pricing  and  ordering  information  is  posted  on 

GAO’s  website,  http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077, 
or  TDD  (202)  51 2-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
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Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our 

Podcasts. 

Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202) 
512-4400,  U.S.  Government  Accountability  Office,  441  G  Street 

NW,  Room  7125,  Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512- 
4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room 

7149 

Washington,  DC  20548 
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